<html>
<head><meta charset="utf-8"><title>Is scanning incompatibilities efficient? · t-cargo/PubGrub · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/260232-t-cargo/PubGrub/index.html">t-cargo/PubGrub</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/260232-t-cargo/PubGrub/topic/Is.20scanning.20incompatibilities.20efficient.3F.html">Is scanning incompatibilities efficient?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="229679182"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/260232-t-cargo/PubGrub/topic/Is%20scanning%20incompatibilities%20efficient%3F/near/229679182" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/260232-t-cargo/PubGrub/topic/Is.20scanning.20incompatibilities.20efficient.3F.html#229679182">(Mar 10 2021 at 15:21)</a>:</h4>
<p>I was up last night thinking about whether there are inefficiencies in how we find Incompatibilities that are (Almost)Satisfied.<br>
The logick gets called a lot! Once per change, per implication, per decision. And it currently does a scan of <a href="https://github.com/pubgrub-rs/pubgrub/blob/dev/src/internal/core.rs#L101">all incompatibilities</a> to find ones that are related to the changed package and then looks to see the <code>relation</code> with the <code>partial_solution</code>. I had lots of thoughts of different data structures to try, but I may have been dreaming so they may not make enough sense to be worth writing down. What I did want to record ( for my reference or in case you had thoughts ) where 3 places where we may be doing more work than needed.</p>
<ol>
<li>Does it happen that we add the <em>same</em> incompatibility to the history multiple times? For example <code>a depends on b</code>, when first we decide on <code>a</code> we will add it to the history, then if something else depends on <code>b</code> then <code>current_package</code> will be <code>b</code> and <code>a depends on b</code> is still <code>AlmostSatisfied</code> so we will add it again.</li>
<li>If we have an incompatibility with one term, how does it get into the history? Does it successfully get re-added after a backtrack? ( this must work, I just couldn't figure it out when I was half asleep. ) We should make sure that the memory knows not to ask for the range agen after our dependency provider tells us that it is not available.</li>
<li>If an incompatibility is found to be <code>Contradicted</code> by the <code>partial_solution</code> is there any reason to check it again ( until a backtrack happens ).?</li>
</ol>



<a name="229922266"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/260232-t-cargo/PubGrub/topic/Is%20scanning%20incompatibilities%20efficient%3F/near/229922266" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/260232-t-cargo/PubGrub/topic/Is.20scanning.20incompatibilities.20efficient.3F.html#229922266">(Mar 11 2021 at 20:41)</a>:</h4>
<ol start="3">
<li>Just had some time to experiment with it. Removing from consideration incompatibilities that are found to be contradicted can be a perf win. Should definitely come back to it when I have more time.</li>
</ol>



<a name="230157797"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/260232-t-cargo/PubGrub/topic/Is%20scanning%20incompatibilities%20efficient%3F/near/230157797" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthieu Pizenberg <a href="https://rust-lang.github.io/zulip_archive/stream/260232-t-cargo/PubGrub/topic/Is.20scanning.20incompatibilities.20efficient.3F.html#230157797">(Mar 13 2021 at 11:54)</a>:</h4>
<p>It's definitely an area that can be improved, and one of the ways to explore is storing incompatibilities in a smarter way for access. (A map probably).</p>



<a name="230157904"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/260232-t-cargo/PubGrub/topic/Is%20scanning%20incompatibilities%20efficient%3F/near/230157904" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthieu Pizenberg <a href="https://rust-lang.github.io/zulip_archive/stream/260232-t-cargo/PubGrub/topic/Is.20scanning.20incompatibilities.20efficient.3F.html#230157904">(Mar 13 2021 at 11:55)</a>:</h4>
<ol>
<li>I don't remember well enough. I'll have a look at it next week</li>
</ol>



<a name="230157995"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/260232-t-cargo/PubGrub/topic/Is%20scanning%20incompatibilities%20efficient%3F/near/230157995" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthieu Pizenberg <a href="https://rust-lang.github.io/zulip_archive/stream/260232-t-cargo/PubGrub/topic/Is.20scanning.20incompatibilities.20efficient.3F.html#230157995">(Mar 13 2021 at 11:57)</a>:</h4>
<ol start="3">
<li>Is it possible that a currently contradicted incompat becomes almost satisfied after a backtrack and thus will be needed later. I don't remember well enough too.</li>
</ol>



<a name="230158366"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/260232-t-cargo/PubGrub/topic/Is%20scanning%20incompatibilities%20efficient%3F/near/230158366" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthieu Pizenberg <a href="https://rust-lang.github.io/zulip_archive/stream/260232-t-cargo/PubGrub/topic/Is.20scanning.20incompatibilities.20efficient.3F.html#230158366">(Mar 13 2021 at 12:03)</a>:</h4>
<p>Sorry o think we are misusing "term" and "incompatibilities" in our conversation. Incompatibilities are always true, so there is no risk of them being removed after z backtrack since backtracking does not touch incompatibilities, just terms in the partial solution</p>



<a name="230158564"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/260232-t-cargo/PubGrub/topic/Is%20scanning%20incompatibilities%20efficient%3F/near/230158564" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthieu Pizenberg <a href="https://rust-lang.github.io/zulip_archive/stream/260232-t-cargo/PubGrub/topic/Is.20scanning.20incompatibilities.20efficient.3F.html#230158564">(Mar 13 2021 at 12:07)</a>:</h4>
<p>So 2. Is fine since the code that builds the range of potentially valid versions always takes into account all incompatibilities</p>



<a name="230158635"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/260232-t-cargo/PubGrub/topic/Is%20scanning%20incompatibilities%20efficient%3F/near/230158635" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthieu Pizenberg <a href="https://rust-lang.github.io/zulip_archive/stream/260232-t-cargo/PubGrub/topic/Is.20scanning.20incompatibilities.20efficient.3F.html#230158635">(Mar 13 2021 at 12:08)</a>:</h4>
<p>Then again I can't point you exactly to the related code. I'll try to next week if you don't figure it out before</p>



<a name="230165842"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/260232-t-cargo/PubGrub/topic/Is%20scanning%20incompatibilities%20efficient%3F/near/230165842" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/260232-t-cargo/PubGrub/topic/Is.20scanning.20incompatibilities.20efficient.3F.html#230165842">(Mar 13 2021 at 14:23)</a>:</h4>
<p>No rush, I will not have time to work on this deeply for a few weeks.<br>
I did some experiments and checking for <code>1.</code> ( if it happens at all ) was found to not to be faster on the elm benchmark. <code>3.</code> was found to have some ~10% improvements. So worth exploring some day.</p>
<blockquote>
<p>Sorry o think we are misusing "term" and "incompatibilities" in our conversation. Incompatibilities are always true, so there is no risk of them being removed after z backtrack since backtracking does not touch incompatibilities, just terms in the partial solution</p>
</blockquote>
<p>I meant that the <code>partial solution</code> stores the terms implied by some set of <code>incompatibilities</code>. When we backtrack we are shrinking that set ( by removing decisions and the terms implied by them ).  An incompatibility with one term like "no versions available", will be the first to have its term removed as it was only discovered after the most recent decision. But in some sense is always relevant as it does not depend on any decision, it is <code>almost satisfied</code> from the beginning.</p>



<a name="230345045"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/260232-t-cargo/PubGrub/topic/Is%20scanning%20incompatibilities%20efficient%3F/near/230345045" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/260232-t-cargo/PubGrub/topic/Is.20scanning.20incompatibilities.20efficient.3F.html#230345045">(Mar 15 2021 at 13:39)</a>:</h4>
<p>I did some quick analysis of the complexity of the packages in the elm benchmark. And I think it is very heavy on small dependence trees. There are only a handful of cases with &gt;200 incompatibilities. (for reference pubgrub-rs has about 200 dependencies in the lockfile) So it is possible that some things improve our big-O but slow down the elm benchmark.<br>
I am thinking of creating some kind of config file that specifies supsets to use as additional benchmarks. Maybe the most populer, the most complicated, and the ones that error, or something.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>